A Case for Understanding End-to-End Performance of Topic Detection and Tracking Based Big Data Applications in the Cloud

Big Data is revolutionizing nearly every aspect of our lives ranging from enterprises to consumers, from science to government. On the other hand, cloud computing recently has emerged as the platform that can provide an effective and economical infrastructure for collection and analysis of big data produced by applications such as topic detection and tracking (TDT). The fundamental challenge is how to cost-effectively orchestrate these big data applications such as TDT over existing cloud computing platforms for accomplishing big data analytic tasks while meeting performance Service Level Agreements (SLAs). In this vision paper, we propose a layered performance model for topic detection and tracking based big data analytic applications that take into account big data characteristics, the data and event flow across myriad cloud software and hardware resources and diverse SLA considerations. We present some preliminary results of the proposed systems that shows its effectiveness as regards to understanding the complex performance dependencies across multiple layers of TDT applications.